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ABSTRACT 

This research guide is designed to help people who 
are vorking on the development of new educational programs for public 
schools. By outlining some of the major questions and problems that 
relate to research of new programs and by suggesting some practical 
steps, this guide attempts to aid planning groups in the use of 
research techniques. Research requires consideration of how 
information Is collected and analyzed. The researcher considers the 
effect of the research approach Itself on his conclusions. Before 
developing a research plan* the basic reasons for doing the research 
must be clarified. Once the overall questions of the research are 
clear the next step Is to determine what approaches to the 
collection, analysis, and presentation of Information will most 
productively lead to the research goals. The uses, limits, and costs 
In terms of time and money are indicated for each approach discussed 
here as well as some suggestions for Implementing that approach. Once 
the Information Is collected, the analysis of the Information will 
determine the quality of the concluslonsa Much of the best analysis 
of neiT school programs has been done by program participants who have 
had no formalized training In research. The choice of how to present 
research conclusions should be based on your audience as well as the 
purpose for presentation. (Author/JH) 
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70: Dr. Michael J. Bakaiis 

Superintendent of Public Instruction 
State of Illinois 



Dear Or Eakalis: 

In partial fulfillment of the terms of our contract with the Office of the Superin- 
tendent of Public instruction. Center for New Schools respectfully submits Resea rch 
for New School Programs; A Guide for Progrorn Planners, 

This guide was developed on the premise that aiding planning groups to develop 
competencies in research will facilitate the process of Improving the quality of public 
education* We are convinced from our work that local community planning groups 
hove the obilities to carry out productive research. They need to learn some specific 
methodological techniques and need to be encouraged to step Into areas that they fear 
belong only to the experts* We hope that this guide serves that purpose for both the 
Affiliates of the Illinois Network for School Development and for other local groups 
in the state concerned with improving education. 

Sincerely, 



Richard Johnson 
Donald R. Moore 
Thomas A* Wilson 

Executive Associates 
Center for New Schools 
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I. INTRODUCTION 

This research guide is designed to help people who are working on the development of 
new educational programs for public schools. It has not been designed to be o comprehensive 
guide to researchers. Rather, by outUning some ol the major questions and problems that re- 
late to research of new programs and by suggesting some practical steps, this guide attempts 
to aid planning groups in the use of research techniques. 

it is written from the Center for New Schools extensive experience in helping plan, 
implement and research new school programs. We have worked with both the theory and 
application of educational research techniques within the context of planning and implement- 
ing new educational programs. This work includes a comprehensive research project as part 
of our work in helping to establish an alternative urban public high school, and several more 
limited evaluation projects of new school programs in the Midwest. Our work has been 
supported by the University of Illinois, the Notional Institute for Education, the National 
institute for Mental Health, and the Carnegie Foundation. 

Our approach to research has been developed to aid the process of setting up and then 
continually improving a new educational program. We became convinced through our 
work that much of the traditional wisdom about educational research was not very helpful to 
progrom planners. We think it is important that research be designed to meet the needs of 
the planners as opposed to the research theoretician. We hope that this guide will help 
program planners by raising some of the issues that we have encountered in our work and by 
suggesting some pracMcai steps about research that we have found effective. 
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II. MAKING RESEARCH USEFUL: DECIDING WHAT YOU WANT TO KNOW 



Research ts not a mysterious, unusual process, in our everyday life it !s usually 
enough to soy you know something, e.g., "Everyone says the new principal is really 
good. " Research methodologies peimit more specific conclusions. "Eighty per cent 

of the parents believe that the principal is doing a good {ob because (l ist of reasons) . 

The other twenty per cent had these criticisms: (list of criticisms.) " Further, 
research gives you the advantage of being able to clearly indicate how you know what 
you know. If a skeptic asked a researcher how he knew about the parents he could re- 
spond, "We interviewed a representative sample of parents in the school. Here are 
the questions we asked and here is how we analyzed the responses we received. " 

Research then requires consideration of how information is collected and analyzed. 
The researcher carefully outlines his approach to collecting and analyzing information 
before he begins and keeps careful trock of any changes in his methods. Further, he 
considers the effect of the research approach itself on his conclusions. Because he makes 
his assumptions and plans expllcitir and clear, another person can follow the process and 
hopefully arrive at similar conclusions. This helps to assure that the infomnation and 
conclusions are not limited to the perception and understanding of one individual. 

Research plans are made before the work begins. This insures that a carefully 
considered plan Is followed. In practice, the process of completing a research project Is 
complex, and untidy. Sometimes it is necessary to change the initial plan due to unforeseen 



problems orliv* new, more Important questions to be researched surfoce In the process. 
Nevertheless, the ability of the reseorcher to explain clearly to others how he knows what 
he knows is basic* 

Before developing a research plan, the basic reasons for doing the research must be 
clarified. Much useless research might have been avoided If someone hod seriously asked 
the researcher before he started "Why do you want to know that?" or "if your project 
works out OS you plan, what good will it do?" 

The questions for which answers are wanted and the approaches for collecting the In- 
fonmotlon to answer the questions must be judged in terms of the basic purpose for doing the 
research. Deciding the purpose for collecting the Information will directly shape the decisi 
about how to collect that Information, The following examples Illustrate how the research 
purpose affects the whole plan. 

1 . Proving It works. One group of planners working on a new 
program to increase students' reading ability decided the thing 
that they wanted to know most was the effect of their program 
on students' reading scores on standard tests. They selected a 
series of reading tests which were administered to the students 
bc'fore entering and after completing the program. They com- 
pared the students' scores to see if some had Improved and were 
able to judge the program's success. 

2» Selling the progrom t o a board of education . Many program 
planners have to prove the validity of their program to on 
outside group which will affect or control the program, such 
OS a board of education. One such group decided that the 
board would listen most closely to progrom information if the 



research was conducted by an educator whom the board {udged 
to be legitimate. A prestigious local educator was hired. He 
started, at the group's suggestion, by asking the board members 
what questions they would like to hove answered. He prepared 
a careful oral presentation of his finding^using an overhead pro- 
jector. The content of the report was designed to directly answer 
the questions board members had raised. 

3. Resolving conflicts over issues In the schools . The staff at a 
certain school was divided as to the advisability of mixing new 
entering students with the old students, or setting up new groups 
only of new students. After severol meetings where the issue 
was batted back and forth It was decided to find out what the 
students thought. A brief questionnaire was designed and a 
representative group of students were polled. The results of the 
poll were reported back at a staff n.seting and the decision was 
made. 

4. Dealing with long tenw problems in the school . Several staff 
members at another school were concerned that teachers and 
administrators we^e not taking proper responsibilities in the 
disciplining of students. An interview for all of the staff was 
developed by the concerned staff members. Teachers were 
asked to lUt specific incidents where they had seen problems 
in staff response to students'behavior. Each teacher was then 
privately Interviewed by a person from outside the staff. The 
infomiatlon was presented to the staff in the form of case studies 
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bullf upon the specific incidents that staff people had identified 
as significant. The case studies were presented in a workshop. 
The resulting discussion on expectations about student behavior, 
control, and staff responsibility was productive. Instead of the 
usual challenges or righteous generalizations which freauently 
characterize this subject, a fruitful discussion was held dealing 
with specific problems of discipline in the school . 
5. S upporting long-term planning . A planning group designed a 
school program to meet certain well specified goals. They 
wanted to know how this program affected the students in 
terms of those goals and to learn about the social and educa- 
tional processes that led to the results. The group thought 
several different approaches to collecting information would 
achieve the best results. They developed questionnaires about 
students' perceptions of school and of the designated goals. 
The questionnaires were administered to students before they 
entered the program and after they had been in the program a 
year. The planning group hired a participant-observer to be 
in the school on a half time basis to carry out a careful obsen/a- 
tion in the areas of interest. An interview format was developed 
which pennitted short interviews of a small number of students 
representative, by race, sex and age of the total student body. 
These interviews were administered monthly. They developed 
a process so ^hat the information would be available to them in 
both raw and analyzed forms, allowing them to deal directly 
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wfth it tn their planning. The Infomiation was used in planning 
sessions which involved not only the original planners, but also 
program participants. 

From these eiCamples the effect of the reseach goals on the methods used to collect and 
analyze infomiation Is clear. Most school research programs combine several of the purposes 
presented in these examples. 

While careful advance planning of research projects is crucial, planners should not be 
inflexible if they see that their initial approach needs to be modified to better reach their 
prime research goals. In some cases when the research is designed as an experiment in itself 
such changes or modifications cannot be made because they would destroy the basic design 
of the research. 

Researchers should consider the possibility that they may learn something that they had 
no intention of learning in their planning, often referred to as "serendipity". Serendipitons 
findings do not result from a iaissez faire approach toward research, but from plans which 
ted to in depth examination of crucial p<irts of the program* 

Advance planning of research pro|ects is difficult work. Deciding on program goals, 
on what the purpose of the research is and on how best to meet that purpose is hard both be- 
cause people will have different ideas about each issue and because the thinking is abstract. 
This process can be more productive if program planners visit other programs and talk with 
people who hove actually worked on new program research. Two guides which might help 
are It Works This Woy For Some; Cose Studies of Fifteen Schools and Planning f or o Change; 
A Res ource Catalogue . They were prepared by Center for New Schools under contract with 
the Illinois Network for School Development and ore available from the Network the office of the 
Superintendent for Public Instruction, Springfield, Illinois, 62706. 
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111. SEVERAL APPROACHES TO COLLECTING INFORMATION 

Once fhe purpose and the overall questions of the research ore clear the next step is to 
determine what approaches to the collection, analysis and presentation of Information will 
most productively lead to the research goats. The following sections discuss approaches to 
the collection, analysis and presentation of information* This section presents several 
approaches to the collection of infonmotion. The uses, limits and costs in terms of time and 
money are indicated for each approach as well as some suggestions for implementing that 
approach . 
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Achievement tests measure a student's progress !n a course of study or academic skill. 
Many achievement tests are designed by teachers to test how well their students have learned 
a lesson or a series of lessons. 

Standardized ochlevement tests have been carefully developed by professional researchers. 
They are standardized in the seose that the same questions hove been asked of many different 
students. Uniform questions provide comparison between the tested students with all students 
who have ever taken the test. Standardized, therefore, means that the questions ore the 
same and that a great deal of information is ovailoble as to how well a student at a certain 
grade level or age has done in the past on the test. When a study indicates the reading grade 
level of students, standardized tests have been used. 

1. Uses: 

- Students' achievement is crucial to most educational programs, thus its 
measurement is important to the research of educational programs. 

- Standardized tests provides a measure of learning on a very carefully 
specified criteria, such as the ability to odd, or the ability to read 
certain kind of material . 

- Achievement tests^nswers, particularly those from standardized tests, 
are easily translated into numbers that con then be treated statistically. 

- Standardized achievement tests pemiits comparisions between different 
students and between the some students over a period of time. 

2. Limits: 

- Achievemenf fests are based on a limited definition of learning. The 
method of testing, which is usually reading a passage and rhon answering 
qucsfions by writing or checking boxes, limits the definition. For example, 
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a person who has learned auto mechanics by watching and doing, but 
has not learned to read cuto manuals or written instrucuons, would get 
a low score on an achievement test because he was required to read* 
in fact, his reading skill and not his actual skill in mechanics was being 
tested* 

- Standardized tests are inflexible in terms of content because they are 
not designed for the goals of a particular learning program. Rather they 
are designed to be standard in their content, and therefore may not be 
able to test the specific skills for which an experimental program was 
designed* 

- Standardized tests were not designed to be sensitive to individual changes 

in performance over a short period of time* in fact they have been designed 
to be stable over short periods of time. Questions that might measure 
changes have been systemati colly wooded out of a standardized test. 

- Success on standardized tests is viewed as very Important by many teachers, 
students and parents. Consequently, the use of these tests often influences 
how teachers teach and how curriculum is designed. 

Costs ; 

Achievement tests ore cheap* Standardized tests which must be bought 
cost only a few cents apiece. Costs for scoring and for analyzing standardized 
tests are also low because procedures hove been carefully developed and often 
are available on a computer. Frequently, the cost of the test includes fhe ex- 
pense of scoring and analysis of tests results. Finally, tests are administered 

« 

cheaply <ind a feacher or a parent can administer them wi^h liftle training. Un- 
fortuanrely, because of their low costs many groups are using tests even though 
they may not be appropriate for their particular research goals, 

13 
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iuggestlons for u sing achievements tests ; 

Because the nature of the test itself detennlnes the information that will 
be collected, the test is key. Many tests are available and it would be advisable 
to locate those tests that most clearly reflect the goals of the program. The regular 
school system testing program should be evaluated in temw of program and research 
goals. Before selecting a test, read and complete the test as if you were a student 
taking it. The actual content of a test may differ from your impression of what the 
test measures from the description of the test. Consider if the questions are appro- 
priate not only to the learning goals but also to the students being tested. 

- Achievement test infonnation is probably the most effective tool for 
convincing outside groups such as boards of education that your pro- 
gram is worthy. Test scores are often considered "more objective " 
than other types of infomiation. The difficulty that this presents for 
program Implementers is that most achievement tests are designed not 
to be influenced on the short temi by programs. 

- Frequently achievement tests are perceived by students as frightening 
and/or mysterious because of their importance in detemiining what 
happens to students during their schooling. We think it is crucial that 
tests not be used to detenmine a student's future. All students who take 

a test should have the results explained to them very carefully, including 
the deliniation of the limits of the tests. 
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III. B. QU ESTIONNAIRES |EST COPY AVAIIABLE 

Questlonnoires ask people fo write answers to written questions. Answers can often be 
written in the respondent's own words. More often, however, the answere are Included as 
one of a number of choices to be checked by the respondent. Questionnaire responses are 
used to record opinions, perceptions, and attitudes as well as to collect simple objective 
information such as "How many students are enrolled in Wilson High School?" Questionnaire 
scales have been developed by psychologists to measure different personality attributes. 

1. Uses: 

- Questionnaires are used to collect specific information that can be 
written down easily. 

- They can be designed so that numbers can easily be assigned to the 
information collected. Thus, questionnaires con facilitate easy 
computation of information. 

- Questionnaires can provide initial information from people which can 
then be used as the basis for discussion or an interview. 

2. Limits ; 

•* Since there is no built in check when a person is filling !n a question- 
naire, it is more easy for a person to respond with what he thinks is 
wanted rather than how he really feels or thinks. 

- Written words are frequently interpreted differently by different people. 
Since there Is no way to check if the person answering the questions 
mean5 the some thing as the person who designed and is interpreting the 
answered questions, gross misinterpretation con occur. Studies have b«-*Mi 
done on the results of questionnaire information by asking the people who 
filled in the questionnaires what their answers meant. It was found that 
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what they thought their answers meant were significantly different 
from what the person who made the questionnaire interpreted their 
answers to mean • 

- Because numbers can be easily given to questionnaire information, 
it is often thought that questionnaires are more objecMve than that 
colJepted by other means. Unfortunately, the act of simply assigning 
numbers to information does not indicate how ob{ective the informa- 
tion is. 

- The infomfiation they provide is limited to written responses which 
often is inadequate to deal with the complexity of educational and 
social processes. 

Costs: 

Collecting information by questionnaire is inexpensive compered 
to other methods. If the questionnaire is standardized, coding and scoring 
schemes are usually available and also inexpensive to use. Developing 
coding schemes and anal/sis of non-standardized questionnaires can be more 
costly. 

Suggestion s for using questionna i res; 

It is essential that the instructions, the questions, and the choice of 
answers are written clearly and simply. The way to Insure this is to sample 
test the questionnaire before using it formally. Have several people who 
will not be given the questionnaire; but who are similar to the intended re- 
spondents, complete the questions. When they have finished, osic if there 
were ony difficulties in understanding the questions. Then you should go 
over each question with the trial taker to see if his understanding of the 
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questions are consistent with the meaning intended. Af^er this trial run, 
revisions are usualiy easily niade. 

The system for collecting and coding the information should be 
developed simultaneously with the questionnaires. This insures that the 
questions will provide the information sought ond that the answers can be 
easily treated in analysis. 

Your first questionnaires may often be too long and ask for un- 
necessary information. Sometimes it is good to ask for infoimation that 
doesn't directly relate to the project because it is cheaper to collect in- 
formation at one time, rather than to keep going back. Be careful that 
such extraneous information does not detract from the collection of infomia 
tion that you desire. Generally the shorter the questionnaire, the more 
informative the response. 
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III. C. I NTERVIEWS 

Interviewing techniques are refinements of a common way of collecting infonnation, 
one person asking questions of another. Interview questions vary from closed to open-ended. 
A closed question is one which requires a straight forward, Infoimational response, for ex- 
ample, "How many teachere are on the staff of this school?" While an open ended question 
usually requires a lengthier response oft^n about perceptions or opinions, and shapes the 
next question in the interview, for exampl«9, "How is this school different from other schools 
you have gone to?" "Tell me more about (one of the difference mentioned) ." 
1. Usest 

- Interviews can collect through oral communication a wide variety of 
information. 

- Interviews are particularly useful to gather in-depth information on 
personal opinions, knowledge or perceptions. 

- While interviews are more costly to administer than questionnaires, 
they have several advantages over them: 

i. An interview is usually taken more seriously by the 
respondent, because he is talking to another person 
rather than filling out a form on paper, 
ii . Because the person answering questions can also ask 
questions to clarify meanings, misunderstandings about 
meanings are less likely to occur, or are able to be 
cleared up, 

iii. Because the interviewer is directly on the scene he is 
more likely to pick up additional, useful information 
from the respondent not covered by the questions. 
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2. Umits ; 

- The location of the interview affect Information you are able to 
collect. An interview conducted in a high school corridor between 
classes will differ from one conducted in private. 

- The interviewer himself will affect the information. Differences in 
sex, age, and status are important. A high school girl interviewed 
by a mole principal will answer some questions differently than if she 
were interviewed by another girl student. In addition, personality 
differences have a subtle but distinct effect. Some of the effects of 
the interviewer such as tone of the interview, and how much to follow 
up an answer with another question can be minimized with careful 
training. 

- Because interviewing is costly and time consuming, only a small 
sample of people are usually interviewed. Thus, the selection of 
those interviewed can be a limitation on how representative the in- 
formation is. 

- Infomiation collected In response to open questions is often difficult 
to code and analyze. 

- Putting interview infonnation into written form is usually a problem. 
Transcribing tape recordings of interview is laborious ond costly. 

Several of these limits can be eased or eliminated through pre-planning. 
(See "Suggestions for using interviews" below.) 

. Costs; 

Interviewing is more expensive then either testing or questionnoires. 
The greater costs are because: 

19 

- The interviewing process itself is time consuming. 
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- Interviewers require careful training. 

- Collecting and analyzing information is usually more complex. 
Sugge stions fo r tising interviews; 

«- Selection of people to be interviewed. 

Because of the cost of interviewing, a researcher will usually 
interview only a select number of people from the group he is studying. 
It is essential that his selection is representative of the iai^er group, 
in order for generalizations to be made about the larger group's under 
study. The best method of selecting the sample for interview is by 
random. Two ways this can be done are (1) drawing names out of a 
hat, or (2) assigning each person in the large group a number. The 
first people whose numbers are listed in a random number table (avail- 
able at the back of most textbooks on statistics) are then chosen. 

Sometimes a researcher will want to insure that certain subgroups 
(e.g., boys and girls, or young and old students) are fairly represented 
in the group to be interviewed. To accomplish this the total group 
should be divided into the sub-groups and the determined number from 
each subgroup should then be drawn randomly. The total group to be 
interviewed will then have the sex or age representation guaranteed 
that was desired. Technically this is called stratified random sampling. 

- Matching interviewer with interviewee. 

Finding interviewers who ore similar to the people being interviewed 
in temis of sex, ethnic background and age will facilitate the interview 
and provide more information than not matching. Consider using porents 
to interview porents and students to Interview students. If the inten/iew 
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training program is sufficienj .matching will probably maice for 
lack of experience. Moreover, using inexperienced interviewers whom 
you train is usually cheaper than hiring professional interviewers. 
- Troining, 

Training of interviewers Is especially important owing to the large 
potential effect^the interviewer on the responses of the interviewee. 
The training session should include the following issues: 

Qu estions to be asked . The interviev4r must understand 
both the general purposes of the questions and the specific word- 
ing, so he can handle queries from the interviewee. 

Mechan ic s of th e interview . Consistency in interview 
setting is necessary. The interviewer should be responsible for 
finding the room and arranging chain and tables and using a 
tape-recorder if required. The plan for scheduling the interviews 
and the scheme for taking notes should be understood* 

The tone of the Interview. The amount of formality versus 
the allowance for the interviewer's penonal style should be clear. 
The amount and type of follow-up questions to responses given 
should also be clear. 

After the training discussions on the questions, the mechanics, 
and the tone of interviews, each interviewer should complete a trial 
interview using the actual interview questions with a person who 
shares the characteristics of the people to be interviewed. This trial 
Interview should be recorded, and then listened to by the interviewer 
and trainer and possibly by the group of interviewers. The recorded 
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Intervlew should first bo listened to without Interruption and then 
played again to be interrupted by the trainer, interviewer or 
other intervievers to ask questions, moke points, and to note diff i* 
cuitles end inconsisrencies with the agreed upon format. This trial 
interview is the most crucial step in the training process. 

Taking notes . Probably the most frequent method for re- 
cordir\g the answers of an interview is to tape record the session. 
This is particularly important if the tone of the interview is important 
to the infomiation collected, such as responses to fndepth personal 
questions. The difficulty with tape recording !s that someone has to 
listen to the tape to get the infonnation . That is a time consuming 
process. A method that loses some of reliability and verifiability 
in terms of insuring that the information analyzed is the same as that 
given by the respondent in the interview is taking notes during the 
interview. One system that has worked well for us is to take notes 
of key points and phrases used by the respondent. The note taking 
process !s explained clearly to the respondent before the interview 
begins. Immediately following the interview the Interviewer spends 
a time equal to the time of the interview in writing up a detailed 
description of the respondent's answers. We frequently use five by 
eight cards writing one response per cord. This permits easy handling 
of the responses later when the responses are analyzed. If the tone of 
the interview is judged to be particularly importont, both note-taking 
and tape recording ore suggested. In this way tapes could act as a 
cross check for the notes and would also be available for further analysis 
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Dev elopment of the questions . While the interviewer can 
correct misunderstanding rising from the questions during the inter- 
view, the construction of the questions themselves should receive 
careful attention. The best questions are brief and clear. 
- Effect of interviewing on the program. 

Since interviewing is often a fairly intense experience, its effects 
on the program should be considered in developing plans. Depending 
on the purpose of the interviewing and on how it is corrled out, its 
effects can be either positive or negative. For example, when the staff 
of a program was interviewed in a secretive manner by a representative 
of the agency funding the program, the unxiety within the program was 
very high and dysfunctionr' ' program goals. In another program, 
after careful explanation of the purposes of the interviews, students 
were inten^iewed to get their Impressions of the program and what needed 
to be improved. The fact that students were asked for their opinions 
about their program, made them feet important and more Involved In the 
program. While the effects of an Interview, or of any information collec- 
tion process can not^and perhaps should not be eliminated, the best 
atmosphere will be created If the researchers explain openly their purposes 
to the program participants and Involve the participants as much as 
possible in the development of the research procedures. The methods 
used for selecting respondents should be very clear to all program portici- 
pants. 

BEST COPY AVAILABLE 
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III. D. OBS ERVATION 

Observation of social events has been used a great deal by onthropologlsts as a research 
technique. Its application in educational settings is quite recent. Despite its drawbacks as 
a means of collecting information about educationol programs, it has proved to be a very pro- 
ductive approach in gathering information to help develop new programs. Its weaknesses stem 
from both inherent difficulties in the approach itself and its relative lack of application in 
schools. 

1. Uses I 

- Observation con provide detailed information on such issues as classroom 
climate, teacher-student interaction, informal social groupings, and 
other social and educational processes. 

- Since the school itself is the actual focus of observation, infonnotion on 
how people actually behave as opposed to how they report they behave 
can be recorded. Such information comes directly from the actual events 
of the school rather than on artificial circumstance created for the pur- 
pose of collecting information such as an interview, 

2. Limits; 

- The observer has a great deal of control over the infomrtation collected in 
that his report is on interpretation of what he sees. For this reason many 
researchers criticize observation as a subjective rather than an objective 
method. Faulting observation for not being objective raises the question 
of is there any methodology which does not warp the phenomena under 
study, ond^whot is "objective" infonnation. Probably a better way to 
evaluate a methodology is to ask if it collects the kind of information 
you need ond what the limits on such Information are because of the way 
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it was coitected. Each methodology has Its purpose and Its limitations* 
Because the person collecting the information does exercise a good deal 
of control over the information which he reports, much consideration 
has gone into the amount of structure that should be involved In the 
observation process. Some observation schemes require the observor 
only to check a list of possible behaviors such as, listening, asking 
questions, and giving answers of what a specified student Is doing every 
two minutes. Other approaches are much more flexible and open ended, 
requiring that the observor develop hypotheses based on his prior experi- 
ence with the situation, and then test out the hypotheses with further 
observation. The open-ended approach has proved more valuable in the 
development of new programs. Open-ended observation is usually called 
"parti ci pant obse rvat Ion " . 

- Observational information is generally very complex and difficult to 
analyze statistically. 

- The success of observation programs depends on the training of the ob- 
servor, his or her ability to gain the cooperation and trust of the people 
under study and the analysis of the Information. Thus, observation is 
more complex in its reiiobllity or verif lability than othe; iechniques of 
collecting infonnation. 

BEST copy AVAiUBLE 

3. Costs ; 

Observation is expensive, its costs include the extensive amount 
of time required to carry out effective observation, and the time to analyze 
complex infonnation. In spite of the costs, school programs which have 
used observation found if on effective rncon of collecting useful Information, 

o 
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Suggestions for using obsetvaHon; 

- Due to the compiexfty and high costs of the observation process, it is 
important for the program participants to understand the reasons and 
values for using this method • The direct involvement of the participants 
in the development of the plans for observation will help towards this 
end. 

- The selection of the observors is key, which is a difficult process. 
Graduate students, particularly in anthropology, represent one likely 
source, parents and community people are other possibilities. Training 
local people is advantageous in that they are more likely to be around 
longer and to hove a definite long-temn commitment in the program. 

The following suggestions apply to the use of observation check 
list approaches: 

- The description of what is being observed needs to be described 
precisely to insure that more than one person would score the 
some event the some way. 

- One to two trials should be run to insure that observors under- 
stand the instructions consistently. During the trial one event 
such OS a class should be observed by all of the observors. Dis- 
cussion should follow the observation so that problems can be 
identified and corrected. 

The following suggestions apply to the use of participant observation 
check list approaches: 

- The selection of the participant obsen<or is particularly important, 
since he or she hos a strong influence on the collection of the 
information. The participont-observor's role and work should be 
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thoroughiy discussed with program participants and the obsen/or 
before beginning the observation. 
- The observer and participants should be aware of the importance 
and difficulty of the observor's role and perspective that com- 
bines the perspective of the program participant with that of an 
objective outside perspective. This Is achieved in part by the 
observor starting his work with as few as possible preconceptions 
about whet the important issues are to focus on. After a brief 
period he should begin to formulate Issues which could be dis- 

■■i 

eussed with the program participants or with a representative 
group of participonl-s with whom he is working closely. After 
discussion of these Issues the observor should be asked to leod 
several feedback workshops on his observations. Effective parti- 
cipant-obseri'atlon combines careful observation over a signifi- 
cant period of time with constant efforts to clarify or verify his 
interpretations with the interpretations of the participants. 



27 



24 



asr copy i^y^iy^Q^ 

IV. ANALYSIS 

Once the Information is collected, the analysis of the information will detemiine the 
quality of the conclusions. Much of the best analysis of ndw school programs has been done 
by program participants who have had no formalized training in research* The following 
suggestions might help: 

1 . Plan your approach to the analysis. It is often necessary to deviate from 
your plans due to new considerations and unfotseen problems in the data 
that weren't anticipated. If there is ambiguity or a change in plans during 
the process, it is important to be able to explain the process by which the 
conclusions weiv reached at the conclusion of the analysis. 

2. Check to insure that the analysis will yield the information that is wanted, 
3* If it makes sense to number the data consider how you are going to use the 

numbers. In riKiny cases percentages or people responding in one way com- 
pared to percentages responding in another is adequate. If the infomnation 
needs a more sophisticated statistical approach, try to find a person well 
trained in statistics who is interested in the purposes of the program and 
who is willing to train a program participant to perform statistical analysis 
in the future. Any work with a consultant should result with the program 
participants having greater capability to handle what the consultant was 
hired to do. 

4. When a conclusion is reached about the information carefully recheck the 
information to see how the conclusion is supported or challenged by the 
data. Be wary of the limitations of the information and the techniques 
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gathering that information when you are developing your conclusions. 
Carefully consider alternative explanations for any pattern that seems to be 
present in the data. For example, an Increase in reading scores explained 
best by something that happened in the program or by how the students were 
selected for the program. 

While simple explanations of data are often the best, remember that the 
program is a complex set of interrelated activities and personalities. Ex- 
planations that isolate one piece of the program are generally weaker than 
those supported by the context of the whole program. 
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V. PRESENTATION 

The choice of how to present reseorch conclusions should be based on your audience 
OS well as the purpose for presentation. Usually, research conclusions are presented in 
a weighty written report. Frequently written reports are required only for program continu- 
ation and funding. The format preferred by that agency should be explored. But such 
reports are not particularly useful for other purposes and other methods of presentotion 
should be considered. The following forms of presentation have been used and might be 
considered: 

1 . Articles and brochures . If the purpose of the presentation is to infonu 
others about the program, articles might be written based on part or on 
the total results of the research. Imaginative use of graphics, illustra- 
tions, and/or pictures will aid the effectiveness of the article. 

2. Other media. If the purpose of the research is to aid others in the develop- 
ment of new school programs, different visual methods of presentation should 
be considered such as film, videotape, and slide-tapes, in this case, more 
emphasis on the process and the descriptions and solutions of the issues and 
problems would probably be useful. 

3. Workshops. If the puipose of the research is to aid the development of the 
program in terms of decision-making or staff problems and issues in their 
daily work, the research results should be presented in workshop sessions. 
It is not sufficient to simply present the findings in written form to program 
participants. Workshops allow time for consideration, discussions and 
attempts to figure out the actual implications of the results on the daily life 
of the program. 
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VI. . PUTTING IT ALL TOGETHER 

The following questions are designed to help program planners and participants to 
think through the research process before it Is begun. They also serve to summorize some 
of the major points of this guide. 

1. Why is research wanted? 

la. Who is it for? What purposes will it serve? 

2. What information is desired? 

3. Is the infonnation that is desired consistent with the reasons for the 
evaluation itself? 

4. What are the most efficient methods for securing the rnfomfiation desired? 

5. How will the infonnation be coded and analyzed? 

6. Will the methods for collecting, coding and analyzing the infonnation 
contribute to the purpose for the evaluation? 

7* How will the infonnation be presented, and to whom? Are there more 
effective ways to present the infomiation other than writing a report? 
What purposes and audiences do other means of presentation serve? 
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